Product Sampling for Software Product Lines

The analysis of software product lines is challenging due to the potentially large number of products, which grow exponentially in terms of the number of features. Product sampling is a technique used to avoid exhaustive testing, which is often infeasible. We have proposed a classification for product sampling techniques and classified the existing literature accordingly. We distinguish the important characteristics of such approaches based on the information used for sampling, the kind of algorithm, and the achieved coverage criteria. Furthermore, we give an overview on existing tools and evaluations of product sampling techniques. We share our insights on the state-of-the-art of product sampling and discuss potential future work.

This website is accepted as an official ACM artifact to our publication at SPLC'18 [1]. As the large number of algorithms and our detailed classification give rise to a large space, that is hard to explore only by means of a table in the proceedings. Here, we offer an interactive table with filter and sorting capabilities. Furthermore, product sampling is a very active research area and new algorithms are published several times a year. We aim to keep the table up-to-date, but kindly ask you to report any missing literature and wrong classifications to us. Instead of sending an e-mail it is also possible to propose changes by means of a pull request. In the request, you can add the paper to the file of Bibtex entries, which is used to generate this website. Please note that there is a dedicated entry called sampling-tags containing the classification (e.g., feature model as input data or name=NewAlgorithm to specify the entry in column entitled Algorithm). We will review the changes manually and update the website accordingly.

[1] Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammadreza Mousavi, and Ina Schaefer. A Classification of Product Sampling for Software Product Lines. In Proceedings of the International Software Product Line Conference (SPLC). September 2018. To appear.

The table above is the result of an expert survey. More details on the methodology can be found in the SPLC'18 paper. Nevertheless, there are a couple of details (not given in the paper) that we aim to share in the following.

• Keyword Search and Filters: The website enables users to search by any keyword, such as author names or publication titles. Furthermore, it is possible to filter particular columns by one or more entries. Several filters for one column are considered as the union, whereas filters in different columns are used as subsequent filters of the search space.
• Links to Google Scholar: When clicking on the title of a paper we direct to a Google Scholar search for that paper. This search typically leads to the publisher page or other freely available source of the paper.
• Names of Sampling Algorithms: For brevity, we had to omit the names of sampling algorithms in the table in the paper. On the website, those names are specified for each row that corresponds to a newly proposed algorithm. This is particularly useful if several sampling algorithms have been proposed in the same paper, but are classified differently.
• Inclusion of Subsumed Publications: In our paper, we omit papers being subsumed by follow-up papers for brevity. On the website, all papers are shown and classified. This is particularly helpful if someone is aware of any particular paper, but it was subsumed by a follow-up work.
• Further Tags: We created a separate column to include further tags for each publication. Those give more details on tools and evaluations. In particular, we included tool names of proposed tools but also of reused tools. For evaluations, we also list other algorithms that have been evaluated. Using the keyword search, it is now possible to see which publications evaluate a particular sampling algorithm (e.g., ICPL).
• Extension to Future Publications: The table in our paper is fixed now, but the website is planned to be updated in the future. We envision that the research community supports us in this task by referring us to new publications. The original catalog being referenced in the SPLC'18 paper can be retrieved by the keyword SPLC18 (including subsumed papers).